Repository navigation
feat(query): add query_arrow for Arrow results in one call - #238
Conversation
POST /v1/query can answer with an Arrow IPC stream and carries the run id, result id, row count, timing, warning and rollup outcome as response headers. The generated client cannot decode that body or surface the headers, so add QueryApi.query_arrow, which returns an ArrowQueryResult. 429 is retried as in query(). A result too large to send inline (202 with X-Result-Id) is read from the Results API, under the max_auto_rows guard.
| total if total is not None else "unknown", | ||
| query_run_id, | ||
| ) | ||
| if max_auto_rows is not None and total is not None and total > max_auto_rows: |
There was a problem hiding this comment.
nit: Fall back to the X-Total-Row-Count header, or fail closed, when get_query_run fails. (not blocking)
The except ApiException at the top of _follow_arrow sets run = None. total is then None, and this guard is skipped. get_result_arrow then pulls the full result into memory with no max_auto_rows check. In _materialize_full, _authoritative_total falls back to the preview's total_row_count, so query keeps the guard in this case.
There was a problem hiding this comment.
Fixed in 2929b46: the follow now reads the result batch by batch and counts rows as it goes, so max_auto_rows holds even when the run lookup fails. Added a test for that case.
| # names the saved result in X-Result-Id instead of sending partial rows. | ||
| ticket, result_id = outcome | ||
| if result_id is None: | ||
| raise ResultUnavailableError(warning=ticket.reason) |
There was a problem hiding this comment.
nit: Raise an error with a message that fits query_arrow here. (not blocking)
ResultUnavailableError says "Query result is truncated … Re-run with auto_follow=False to use the preview." query_arrow has no preview. This branch also runs before the follow check, so auto_follow=False raises the same error. A caller who follows the advice gets the same exception again.
There was a problem hiding this comment.
Fixed in 2929b46: this case now gets its own message, which names the query run and no longer mentions a preview or auto_follow.
Count rows while reading the followed result, so max_auto_rows holds when the query-run lookup fails. Give the no-result-id case a message that fits query_arrow.
|
|
||
| batches = [] | ||
| fetched = 0 | ||
| with _ArrowResultsApi(self.api_client).stream_result_arrow( |
There was a problem hiding this comment.
nit: Pass limit=max_auto_rows + 1 to stream_result_arrow when max_auto_rows is not None. (not blocking)
The row guard caps memory but not transfer. When the guard raises inside the with block, stream_result_arrow exits through _release_conn, and _release_conn calls drain_conn (hotdata/arrow.py:64). drain_conn reads and discards the rest of the body, so the client still downloads the full oversized result. With the limit, the server sends at most one row past the guard, and the existing fetched > max_auto_rows check still detects the overflow.
What
Adds
QueryApi.query_arrow, which runs a query and returns the rows as apyarrow.Tablein one request.POST /v1/queryalready answers with an Arrow IPC stream when asked, and sends the values the JSON body would carry as response headers. The generated client cannot decode that body or surface the headers, so until now Arrow meant fetching the stored result in a second call.Behaviour
ArrowQueryResult:table,query_run_id,result_id,total_row_count,execution_time_ms,warning,rollup(RollupUsage),followed.dialectand the otherQueryRequestfields work as inquery.RetryPolicyasquery.X-Result-Id) is read from the Results API, under themax_auto_rowsguard. Withauto_follow=Falseit raises the newResultNotInlineError, which carries theresult_id.ValueError.ApiExceptionwith the status, as inquery.Tests
tests/test_query_arrow.py(stubbed transport, real Arrow bytes).tests/integration/test_query_arrow.py. Passes against a live workspace.